1 link tagged with all of: quantization + llm inference + attention + kv cache
Click any tag below to further narrow down your results
Links
@akshay_pachaar on X
A first-principles walk through what happens between your prompt and the streamed reply: tokenization, attention, and why the model does two very different jobs with two different bottlenecks. The piece argues that KV cache memory, not raw model intelligence, is what usually limits long-context speed.
- Generation is a loop that predicts one token at a time, and the first token is slower than the rest because prefill is compute-bound while decode is memory-bound.
- The KV cache avoids recomputing attention over the whole sequence, but at roughly 1 MB per token for a 13B model, a 4K context burns about 4 GB of VRAM.
- Quantization is the highest-leverage knob: a 7B model drops from 14 GB at FP16 to 3.5 GB at INT4, often with only a point of benchmark loss.
- DeepSeek's V4 previews reportedly cut the cache to about 10% of its predecessor's size at a one-million-token context, showing the field is redesigning attention around the cache.